Assessing Semantic Relevance by Using Audiovisual Cues

This paper presents two complementary approaches for assessing semantic relevance in video retrieval—(1) adaptive video indexing and (2) elemental concept indexing. Both approaches make extensive use of audiovisual cues. In the former, retrieval is performed by using implicit semantic indices through audio and visual features. Audio features are extracted by statistical time-frequency analysis that applies Laplacian mixture models to wavelet coefficients. Visual features are extracted by adaptive video indexing techniques that place a strong emphasis on accurate characterization of spatiotemporal information within video clips. The joint audio-visual retrieval methodology is used to search movie clips for semantics such as: love scene, fighting, ship crashing and music video. In the latter, retrieval is performed by using explicit semantic indices. The concept language used to communicate non-verbal information in video is explicated into a set of elemental concepts. Audio and visual cues are then used to detect these elementary concepts from video documents. Semantic queries are then supported by post-indexing coordination of the elemental concepts.